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ABSTRACT 


Restructuring within the Department of Defense since 1990 has forced the 
Defense Contract Management Command (DCMC) to reduce its staffing level from 
24,000 to 14,000 employees. The Contract Administration Offices that make up DCMC 
have reduced from over 100 offices to only 65 offices. This study evaluates the 
efficiency of these Contract Administration Offices in the wake of this massive 
reorganization. Statistical analysis determined the most crucial inputs and outputs for use 
in an efficiency model. Data Envelopment Analysis (DEA) methodology identified 
efficient and inefficient Contract Administration Offices. Further analysis of the DEA 
models revealed potential improvement strategies for inefficient offices and suggests uses 
of the models for future personnel resource forecasting. 
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EXECUTIVE SUMMARY 


The Defense Contract Management Command (DCMC) has undergone a 
significant reorganization in recent years. The number of employees within the 
command has reduced from 24,000 in 1990 to 14,000 today. It is unknown, however, 
whether this reorganization resulted in an efficient organization. This study determines 
the efficiency of the 65 individual Contract Administration Offices located throughout 
the United States that are assigned to DCMC. 

Part of the Defense Logistics Agency, DCMC provides contract management 
services for most contracts awarded within the Department of Defense; over 350,000 
contracts managed at any given time. In order to manage the implementation of these 
contracts. Contract Administration Offices (CAO) provide continuous contract 
management services for government program managers overseeing individual 
manufacturers under contract. CAOs provide support for fact-finding and negotiations 
between contractors and contracting officers, provide safety and environmental 
assurance, and evaluate contractor processes and progress, among other duties. Two 
types of CAOs operate within DCMC: Plant Offices and Area Offices. Area Offices 
manage contracts within a geographic area. Plant Offices are resident within contractors’ 
plants. Because of the inherent differences in the two CAO types, management has 
determined that each office type be modeled separately. 

Efficiency is a ratio of outputs to inputs. However, difficulties arise in this simple 
concept when attempting to compare efficiency between units without easily identifiable 
or countable inputs and outputs or when units have multiple inputs or outputs. The 
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methodology known as Data Envelopment Analysis or DEA, uses mathematical 
programming techniques to combine multiple output and input measures to produce a 
single efficiency score for each organization. 

The selection of measures for use in the models is of utmost importance when 
implementing DEA models. Evidence of a causal relationship between inputs and 
outputs must be shown though analysis of their associated correlation coefficients. 
DCMC provided one candidate input measure and eight candidate output measures of 
which only six output measures are selected for use in the models. Of those six, three fail 
to show strong evidence of this causal relationship. However, those three measures, the 
only measures of quality available, are deemed essential to the overall formulation of the 
DEA models, even though their low correlation coefficient with the input measure 
weakens the final efficiency results. 

Fifteen of the 65 offices are shown to be efficient. Inefficient offices can 
determine improvement opportunities by examining the measures of efficient offices. To 
imp rove, an office must either lower its input, increase output, or both. The resulting 
scores show that some offices are achieving a greater number of outputs per input 
compared to other offices. Efficient offices can be used as a benchmark for other offices 
to emulate. 

The DEA models in this study provide a valuable management tool to assist 
DCMC in determining personnel resource requirements. The great added value with the 
DEA models is the new visibility over each office's strengths and weaknesses. The rapid 



identification of these weaknesses allows management to focus on problems and solve 
them quickly without a great time investment searching for possible causal factors. 
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I. INTRODUCTION 


...We tend to meet any new situation by reorganizing: a wonderful method 
... for creating the illusion of progress, while producing ...inefficiency." 

-- Petronius Arbiter, 210 BC 

A. OVERVIEW 

By any measure, the Department of Defense has undergone dramatic 
reorganization in recent years. Less bureaucracy and smaller organizations are always 
assumed to be better and in the best interest of the taxpayers. This also assumes that 
governmental organizations are producing the same services as in the past but with less 
"overhead" or cost, although relatively few studies can support this claim. In fact, little 
evidence exists to support the notion that downsizing and reorganization within the 
defense department has produced more efficient organizations. 

The Defense Contract Management Command (DCMC), part of the Defense 
Logistics Agency, is charged with managing a staggering number of defense contracts. 
Along with nearly all defense related organizations, DCMC has seen its personnel rolls 
drop steadily since 1990. This reorganization should have resulted in a lean, efficient 
command. The efficiency of the 65 Contract Administration Offices based in the United 
States that make up DCMC is the subject of this thesis. 

In its most basic form, efficiency is a ratio of products (or outputs) to inputs. 
Regardless of the type of output and input, resulting efficiency is simply computed as 
outputs per input unit. For example: "Automobiles per dollar" is an effective means to 
compare efficiency between automobile factories producing similar products. 
Difficulties arise quickly when attempting to compare efficiency between units without 
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easily identifiable or countable inputs and outputs. Products from "white collar" workers 
such as managers and clerical staff are far more difficult to quantify. In private 
enterprise, profit is always available for use as the measured output. Because of the lack 
of a profit statement in the public sector, these organizations are often characterized by 
multiple outputs and occasionally multiple inputs (Ganley, 1992, p.2). The very nature of 
public sector efficiency analysis lends itself directly to the methodology known as Data 
Envelopment Analysis or DEA, which combines multiple output and input measures to 
produce a single efficiency score for each organization. DEA is the basis for the 
efficiency analysis of the 65 DCMC Contract Administration Offices. 

B. BACKGROUND 

Defense Contract Management Command (DCMC) provides contract 
management services for diverse products and services for most contracts awarded within 
the Department of Defense. DCMC manages over 350,000 contracts with 23,000 
contractors worldwide totaling over $900 billion in unliquidated contractual obligations 
and employs over 14,000 employees (Defense Contract Management Command, 1998). 

In order to manage the implementation of these contracts after their award, DCMC, 
headquartered in Fort Belvoir, Virginia, is organized into three major districts: Eastern 
District, Western District, and International District. Each District is responsible for the 
control and supervision of several Contract Administration Offices (CAO) which provide 
continuous contract management services for government program managers overseeing 
individual manufacturers under contract to the Department of Defense (Defense Contract 
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Management Command, 1998). In this analysis, only Eastern and Western Districts are 
analyzed. 

A total of 65 CAOs actively administer contracts for the districts in DCMC. 
Contract Administration Offices provide support for fact-finding and negotiations 
between contractors and contracting officers, provide safety and environmental 
assurance, evaluate contractor processes and controls, evaluate contractor corrective 
actions, provide control of government property, and provide an independent evaluation 
of contractor progress (Defense Logistics Agency, 1998). 

In general, both the number of active offices in DCMC and the overall staffing 
level at each Contract Administration Office have reduced since 1990 due to Defense 
Department downsizing. Because of this, proper resourcing of Contract administration 
Offices has come into question. It is not known if individual CAOs are over or 
understaffed with respect to their assigned workload (Boyce, 1998). 

Two types of CAOs operate within DCMC. Defense Contract Management Area 
Offices (DCMAO) manage contracts within a geographic area. Defense Plant 
Representative Offices (DPRO) are resident within contractors’ plants and are 
responsible for administration services with that particular contractor only. Contract 
management includes daily, on-site surveillance of contractor systems and program 
specific concerns that cannot be viewed or monitored by off-sight agencies. The CAO 
organization allows DCMC personnel to reside close to or in contractors’ facilities and to 
tailor their services to specific customers (DLA, 1998). 

Of the 65 CAOs, nearly half are DPROs and half DCMAOs (34 and 31 
respectively). The greatest difference between area offices and plant offices is the size 
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and number of contracts managed. Area offices tend to manage many (sometimes 
thousands) of contracts of limited size and scope. Plant offices manage fewer contracts 
with a specific contractor but each contract generally has a larger value and associated 
complexity. For this reason, measures of production output for these two types of offices 
are quite different. Area offices tend to measure output by quantity of contracts managed 
while plant offices tend to use the size or dollar value of the contract managed. 

Because of the inherent differences in the two CAO types, (different management 
techniques, volume, type, and complexity of work), DCMC management has determined 
that each office type should be modeled separately. The two models are referred 
throughout this text as the Plant Office Model for DPROs and Area Office Model for 
DCMAOs. 

As will be shown in following chapters, the methodology known as Data 
Envelopment Analysis (DEA), a mathematical programming technique, provides a 
methodology to determine efficiency among public sector organizations with multiple 
input and output measures (Norman, 1991, pp. 5-8). When implementing DEA models, a 
great deal of care must be taken when selecting measures for inclusion in the model. 

Data analysis techniques of the potential input and output measures guide the modeler to 
select not only the proper measures for the model, but also the appropriate DEA model 
formulation. 

Once the measures and specific DEA model are selected, an efficiency score is 
computed for each office. These scores not only identify inefficient and efficient offices, 
they also suggest potential improvement strategies for inefficient offices. 
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In Chapter II, a brief discussion of Data Envelopment Analysis and its relevance 
for this study is investigated. Chapter HI determines which potential input and output 
measures the proper attributes for inclusion in a DEA model. Chapter IV shows the final 
formulation of the DEA model and discusses the results as well as insights into the 
sensitivity of the model and potential improvements for inefficient offices. 
Recommendations and conclusions are provided in Chapter V. 
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II. METHODOLOGY: DATA ENVELOPMENT ANALYSIS 


A. EVALUATING CONTRACT ADMINISTRATION OFFICES 

The combined effect of Defense Department downsizing and subsequent 
reduction in defense related acquisition of goods and services has been reduction in both 
the number and size of contracts managed by DCMC. Some CAOs have consolidated or 
closed, sometimes increasing the workload at a CAO, but in general, workload has 
decreased (Boyce, 1995, p. 11). 

Proper staffing of individual CAOs at DCMC has long been a subject of debate. 
No effective means of scientifically determining the proper staffing level existed until 
very recently. In 1995, the Defense Logistics Agency Office of Operations Research and 
Resource Analysis fit a linear regression model of staffing as a function of workload 
(Boyce, 1995). Proper resourcing was then taken to be the value of the predicted staffing 
from the regression line for a particular workload. Two separate regression models, one 
for plant offices and one for area offices were used to determine staffing. A feature of 
these regression models is that a few offices with great influence may induce an improper 
target staffing level (either too high or too low). Using regression also assumes the 
notion that an "average" office exists without taking into account the unique missions of 
different offices. Most importantly, the regression methodology does not clearly identify 
offices that are particularly efficient and establish targets for the remaining offices to 
reach. In fact, the regression model could lead decision makers to add personnel 
resources to an efficient office because its outputs per unit input do not conform to the 
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average. A better response is to identify the efficient CAOs as the benchmark for other 
offices to emulate. 

Determining relative efficiency between these offices a better method for 
comparing their performance. If every CAO produced exactly the same products, simple 
ratios of inputs and outputs could be used to compute relative efficiencies easily. The 
offices can be ranked by their efficiency of outputs / unit input. This metric gives great 
insight into those offices with the best business practices. If one “best practice” office 
can gain two outputs per unit input, surely all offices should strive to attain that goal. 

However, a fundamental problem arises when more than one input or output is 
necessary for relative efficiency comparisons. It is possible that an office may produce a 
lot of one output but relatively little of another. A composite efficiency score can be 
computed by taking the mean efficiency score. Alternatively, management could assign a 
weight or relative importance to each output or input, thereby developing a weighted 
average efficiency for each office. Unfortunately, both of these methods impose certain 
assumptions on the underlying data. In the first case, each output and input is assumed to 
be equally vital to each office’s overall efficiency. In the case of management 
determined weights, the assumption is that those heuristically developed weights are 
valid and that they apply to every office. The “one size fits all” approach is not always 
appropriate. In particular, for organizations that may have different contexts or 
environments in which to work, there may be great variability among output and input 
measures. 

What truly is required, in order to find a fair measure of an office’s efficiency, is 
an efficiency score that is optimal for that office: an efficiency score which has the best 
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weight on each output and input so that an individual office achieves the highest possible 
score. This provides each office with an evaluation of its inherit efficiency based on its 
strengths. This is exactly how Data Envelopment Analysis computes relative efficiency 
scores with multiple inputs and outputs and why it offers an excellent methodology to 
determine the efficiency of Contract Administration Offices. 

B. DATA ENVELOPMENT ANALYSIS 

1. Overview of the Methodology 

Data Envelopment Analysis (DEA) is a mathematical modeling methodology that 
determines a Decision-Making Unit’s (DMU) efficiency using non-linear programming 
techniques. (Because each non-linear program is a fractional program, it can be easily 
translated into an equivalent linear program.) Decision Making Units are production 
nodes that transform the inputs into outputs. For the purpose of this research, CAOs are 
the DMUs. 

Originally developed by Chames, Cooper, and Rhodes in 1978 (Chames, 1994, 
pp. 23-25), DEA models calculate an efficient frontier by comparing each DMU’s inputs 
and outputs. The efficient frontier is composed of DMUs that have attained the maximal 
efficiency score of one. All other offices, those that are inefficient, have an efficiency 
score less than one and are not on the frontier. The DEA model determines a relative 
efficiency score for each DMU which represents how far the DMU is from an efficient 
frontier (Norman, 1991, p. 43). 

In order to calculate the efficiency of a particular DMU, mathematical 
programming techniques are used to determine “virtual multipliers” or “weights” for the 
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relative value of the various outputs and inputs that maximize a specific DMU’s 
efficiency score. The weights are thus the primal variables in the mathematical program. 
A particular DMU may utilize any combination of inputs and outputs in order to 
maximize its own efficiency score subject to the constraint that all other DMUs’ 
efficiency scores using that particular DMU’s weights are less than or equal to one. A 
separate linear progr ammin g formulation is used to calculate the efficiency score for each 
DMU. It is important to note that DEA models produce only relative efficiency scores in 
comparison to all other DMUs. Unlike regression techniques that estimate an average 
production function, DEA techniques identify two subsets of DMUs: the efficient and 
inefficient. 

2. Model Formulations 

The two most prevalent DEA model formulations are the model developed by 
Chames, Cooper, and Rhodes (1978), and the model developed by Banker, Chames, and 
Cooper (1984), called the CCR and BCC models respectively. The most simple, the 
CCR, is formulated in Figure 1. 
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Indices: 


i 

Output Types (1,2,...I) 

j 

Input Types (1,2,..J) 

k 

DMUs (1,2,...K) 

Data: 


Xkj 

Value of input j for DMU k 

yki 

Value of output i for DMU k 

z 

A small constant. 

Variables: 

v i 

Weight on input j (unitless) 

w t 

Weight on output i (unitless) 

Formulation: 

For each DMU o, (o = 1,2,...K) 


Y y 0 iWi 

Maximize -=L= - 


2. XojVj 
i 

Subject to constraint: 


X y hWi 

‘ <1, fork = l,2,...K 

2, xm 

j 


Wj>£, for i = 1,2,...I 

Vj> S , forj = l,2,...J 


Figure 1. CCR DEA Model Formulation (Norman, 1991, p. 235) 

The most significant difference between the formulations of the CCR model and 
BCC model lie in how the model accounts for output / input returns to scale. The CCR 
model assumes constant returns to scale, such that an increase in an input parameter has 
a corresponding and constant increase (or benefit) in an output parameter. The BCC 
model accounts for variable returns to scale. This occurs when the benefit in output per 
unit input decreases as the input becomes larger. The basic BCC formulation, given in 
Figure 2, is quite similar to the CCR model except for the addition of a constant, c, in the 
objective function. 
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Indices: 

i 


Output Types (1,2,...I) 
Input Types (1,2,.. .J) 
DMUs (1,2,...K) 


j 

k 


Data: 

%kj 

Value of input j for DMU k 

yki 

Value of output i for DMU k 

s 

A small constant. 

Variables: 

v j 

Weight on input j (unitless) 

Wj 

Weight on output i (unitless) 

Constants: 

c k 

Constant for each DMU k 

Formulation: 

For each DMU o, (o = 1,2,...K) 


^ yoiWi+Co 


Maximize 


XojVj 


Subject to constraint: 

^ ywwi+co 


<1, fork = l,2,...K 




Wj> S 9 fori = 1,2,...I 
Vj > £, for j = 1,2,...J 


Figure 2. BCC DEA Model Formulation (Norman, 1991, p. 247) 


C. CONCLUSION 

Data Envelopment Analysis provides an appropriate approach for determining the 
efficiency of Contract Administration Offices. The first and most challenging step in 
formulating a DEA model is the selection of the input and output measures. The 
following chapter will determine the best collection of measures from a field of available 
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statistics to use in the two CAO models. The measures for each model will be selected 


separately, so that only the most appropriate data is selected for those models. Finally, 
the relationship between output and input measures is investigated in order to select the 
proper DEA formulation: CCR (constant returns to scale) or BCC (variable returns to 
scale). Once the model is formulated, insights into the efficiency of DCMC are made and 
improvement strategies observed. These efficiency scores can be used by management to 
assist in determining the proper resource requirements for these offices. 
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III. MODEL FORMULATION 


Selection of inputs and outputs for Data Envelopment Analysis models is at the 
center of the model formulation. The challenge resides in determining the proper 
measures that most accurately, completely, yet succinctly describe the true effectiveness 
of Contract Administration Offices in the two models to be formulated (Plant Model and 
Area Model). In this chapter, after briefly describing the data, its origin, and some if its 
characteristics and limitations, the relationship of the inputs to the outputs will be 
analyzed in order to determine their possible contribution in the two DEA models. 

Public sector organizations often are forced to rely on subjective, rather than 
objective measures of performance because what may be assumed to be outputs, are often 
assigned requirements, not under the control of the organization. That is, increases or 
reductions in inputs may not affect outputs at all. The effect of reducing an input, while 
increasing efficiency score, may only reduce the quality of service of that particular 
office. In the case of DCMC, a combination of subjective and quantitative measures are 
available for analysis and possible inclusion in the DEA model. 

Chames, in his seminal work with DEA, stated that in general, not more than one 
measure should be included for every three evaluated Decision Making Units (DMU) 
(Chames, 1994, p. 73). (As the number of measures approaches the number of DMUs, 
the proportion of DMUs evaluated as efficient will approach 100%. Each added 
measure, in effect, provides another factor in which the DMU attempts to maximize its 
efficiency score). As the DCMC models contain 31 DMUs for the Area Model 34 
DMUs for the Plant Model, every effort to maintain fewer than 10 measures of inputs 
and outputs in total must be taken. 
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Finally, a positive relationship should exist between selected inputs and outputs. 
As the inputs in the data set generally increase, the expected result is an increase in the 
associated outputs. 

A. DATA DESCRIPTION AND INITIAL MEASURE SELECTION 

A panel of subject matter experts at DCMC chose candidate input and output 
measures for further analysis based on their knowledge of the core performance 
requirements of these offices. They determined that all CAOs perform certain core tasks 
that account for the vast majority of the workload and additionally are evaluated on key 
quality measures. Thus, the candidate output factors measure aspects of workload and 
performance quality (Russell, 1998). 

1. Databases 

Three databases maintained by DLA provide the candidate input and outputs: The 
Performance Labor Accounting System (PLAS), Mechanization of Contract 
Administration Services (MOCAS) database, and the Automated Metric System (AMS) 
database. All data analyzed was collected in fiscal year 1998 and usually includes totals 
for the year. In certain cases (as noted) averages or other measures are used. In general, 
non reported data is given a default value of zero. 

The Performance Labor Accounting System (PLAS) records every hour worked 
by an individual staff member for a particular CAO and records the type of work he or 
she is accomplishing for that hour. Employees, regardless of position, are required to 
record hours worked according to the kind of contract (contract kinds are defined as 
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Systems Acquisition, Research and Development, Maintenance and Facilities, Service, 
Supply, and Subcontracts), and by the type of work (inspections, audits, cost analysis, 
etc.). The PLAS data base enables rapid computation of the number of hours worked at a 
particular office in a given time period (DCMC Unit Cost, 1998, p. 6). 

The MOCAS database tracks contracts by kind and their value. This data base 
provides a count of the contracts managed by each CAO as well as their outstanding 
dollar value (unliquidated obligations). These two measures serve as candidates for 
workload (number of contracts and value of contracts) and may be used as outputs in the 
model, (Defense Logistics Agency, 1998). 

The Automated Metric System (AMS) provides key measures of performance to 
DCMC management on each CAO. Certain key measures that Defense Contract 
Management Command uses for management decisions are the following: Overhead 
costs per hour of mission related work, customer satisfaction rate, contractor on-time rate, 
and the value of savings to the government resulting for actions taken by a particular 
CAO. (DLA, 1998). Each of these measures are candidate outputs for the efficiency 
model. 


2. Candidate Inputs and Outputs 

The DCMC provided ten separate data elements for each CAO. These measures 
will each be explained in this section and are summarized as follows: Hours Worked, 
Performance Based Activity Measure, Customer Satisfaction factor, On-Time rating. 
Supply Contracts Managed Per Month, Other Contracts Managed Per Months, System 
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Acquisition Contracts Value Managed Per Months, and Return on Investment, and 
Overhead Costs per hour. 

a. Inputs 

The total number of hours worked in a year can be further divided into 
component parts by contract kind. Because DEA methodology considers each input and 
output equally, the DEA model may base an efficiency score on a single input measure, 
such as total hours for one contract kind. For this reason, the subject matter experts 
specified that the input measure be total hours worked in a given year. The most 
complete and current data is for fiscal year 1998 (DCMC Unit Cost, 1998, pp. 4,10-12). 

b. Outputs 

Considerable effort has been spent at DCMC in an attempt to properly 
quantify the products of CAO efforts. A difficulty in quantifying products is the fact that 
CAOs do widely varying work based on differing contract kinds and contractors 
overseen. The most common output measure employed by DCMC is simply a count of 
the number of contracts managed in a given month (contracts managed per month or 
CMM). This measure, provided by the MOCAS data base, is derived from the number of 
contracts that have seen activity at the close of a particular month. Using CMM as a 
production unit has obvious drawbacks because it does not take into account the size, risk 
or complexity of a particular contract. For example, contracts for major new systems 
(tanks, trucks, ships, etc.) normally have a great deal of complexity, risk, and associated 
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work where recurrent supply or facilities contracts do not (DCMC Unit Cost, 1998, pp. 
10 - 11 ). 

As a means to partition the data set, contracts are categorized as one of 
three kinds. System acquisition. Supply, and Other contracts. Systems acquisition 
contracts, the most complex and high dollar valued contracts, are not normally measured 
in terms of CMM. Because of their relatively low quantity and high value, they are 
considered as a separate category. Supply contracts are the greatest single subset by 
contract kind in the CMM data set, making up 28% of the total count of contracts within 
DCMC. To provide further insight into components of the eventual efficiency score, 
supply contracts were also separated from the total and used as a candidate output. 

The remaining contract kind, systems acquisition is measured by its size or 
value remaining to be filled for the government. This unfilled obligation is called 
“unliquidiated obligations” in DCMC literature and is measured in units of millions of 
dollars. 

The Return on Investment (ROI) is the sum of all savings to the 
government as a result of actions taken by a particular CAO. Examples of some of these 
actions are price negotiations, production process improvements, refunds, costs 
recaptured through litigation, and compensation for the unauthorized use of government 
property. This output is a direct result of cost saving measures taken by CAOs and is 
measured in millions of dollars in savings (DCMC Unit Cost, 1998, p. 24). 

The Customer Satisfaction rating, provided by the AMS database, is a 
subjective measure of performance graded by purchasing authorities, such as government 
program managers, whenever they request and utilize CAO services. Each office 
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received a score between one and six, where six was a perfect score (DCMC Metric 
Guide, 1998, p. 81). Unlike most other data elements, this measure is an average score 
that was tallied when more than one rating was received in FY98. Unfortunately, the 
reporting rate is estimated to be only 20% (Kuhl, 1998) and fully 12% of all offices have 
no rating. Further, little relation between the candidate input (total hours worked) and 
customer satisfaction exist (as will be shown). However, this is the only measure of 
CAO service quality available; certainly an important aspect of efficiency. Despite its 
shortcomings, management at DCMC feel that the customer satisfaction rating is a 
significant factor to include in overall performance evaluations. 

The On-time rate factor for a CAO is a measure describing each managed 
contractor’s ability to meet production time lines set forth in their contract with the 
government. CAOs influence the on-time rate indirectly by providing timely and 
appropriate services to facilitate the rapid fulfillment of the contract. This is also an 
average rate based on contract deliveries for FY98 (DCMC, Metrics Guide, 1998, p.83). 

3. Environmental factors 

DCMC selected two other data elements as candidates for variables in the model. 
However, neither of these are truly an input or an output even though they may have a 
descriptive nature regarding the efficiency of CAOs. As environmental factors, these 
measures are useful in providing explanations about the behavior of the model and in one 
case, may be included in the model, although not as a unique measure. 

The factor called the Performance Based Activity Measure (PBAM) is a measure 
of the relative risk an individual office must work under (DCMC, 1996, p. 45). To 
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measure this risk, factors such as experience level of contractors, complexity of contracts, 
and administrative overhead for particular offices based on their assigned contracts were 
scored to show that higher risk requires more inputs. Offices were assigned a rating 
between 3 and 9 with higher factor equating to higher relative risk. The implication is that 
offices with higher risk factors require more input for proper and adequate management. 
In terms of the efficiency model, an office with a higher PBAM rating can be expected to 
require more input resources to gain the same quantifiable outputs as offices with lower 
PBAM rating. 

To accommodate this factor in the model, each output can be scaled by the PBAM 
factor (or each input by the inverse of PBAM). The effect of the scaling is to normalize 
the data with respect to risk. Those offices that “expect” to have more inputs for the 
same quantifiable output, have thus been equalized with their counterparts that have a 
lower associated risk. Once the DEA model is completed and evaluated, the PBAM 
scaling factor can be multiplicatively removed to determine true improvement strategies. 

The second environmental factor is a measure of indirect costs per hour for each 
CAO. This can be analyzed as an overhead cost per hour of doing business. Examples of 
overhead costs are hours billed as a result of training or clerical work not related directly 
to contract administration. This factor, too, does not resolve itself as an input or an 
output but is rather a descriptive measure of management practice. Once efficiency 
scores are determined it may serve as a comparative tool to determine if a correlation 
exists between efficient offices and their associated overhead costs or if a “best practice” 
or target overhead cost exists. 
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For simplicity, each of the two candidate inputs, six candidate outputs, and two 
environmental factors are labeled in a brief format from the following table. Each is 
referred to throughout this text by their abbreviated names. 


Type 

Description 

Units 

Abbreviation 

Input 

Total hours recorded by CAO in 
fiscal year 1998 

hours 

Hours 

Input 

Total hours recorded scaled by 
inverse of PB AM factor 

hours 

S.hours 

Output 

Customer satisfaction rating 

unitless 

CustSat 

Output 

On - time delivery factor 

% on time 

Ontime 

Output 

Millions of dollars of unliquidated 
contract value managed per month, 
for systems acquisition contracts 
only. (Total for fiscal year 1998) 

dollars 

(million) 

MUMM 

Output 

Supply contracts managed per month 
(Total for fiscal year 1998) 

contracts 

SupCMM 

Output 

Contracts managed per month for all 
other contract kinds. (Total for fiscal 
year 1998) 

contracts 

OthCMM 

Output 

Return on investment (Total fiscal 
year 1998) 

dollars 

(million) 

ROI 

Environmental 

Performance Based Activity Measure 

unitless 

PBAM 

Environmental 

Overhead costs per billed labor hour 

dollars 

Overhead 


Table 1. Candidate Measures 


B. ANALYSIS OF CANDIDATE MEASURES 

In order to show a ratio of inputs to outputs as meaningful in a relative efficiency 
model, increases in inputs must be shown to increase outputs (Norman, 1991, p. 69). 
Further, if the CCR model is to be used, these relationships must be linear. In each 
analysis, data was partitioned into two sets: One for geographical area offices, and one 
for plant representative offices. 
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1. Analysis Of Linearity 


Figures 3 and 4 show all pairs of plots for the eight measures of Table 1 for Area 
Model and Plant Model respectively. In this case, clear relationships are difficult to 
ascertain in certain instances. However, even those measures with the most scattered 
data points, such as OnTime, do show an increasing pattern with respect to inputs. No 
evidence of non-linearity exists in any of the measures from either of the models. Due to 
some instances of non - reports in the CustSat data and ROI data, these plots tend to look 
bi-modal. Specifically, the customer satisfaction rating tends to be between 5 and 6, or a 
default value of zero if no report is given. When non-reporting activities are removed 
from both CustSat and ROI, a more linear trend appears in the data plots for each model. 


Hours 

o e 

1 

» 

c v o 

c vS£& o 

o" o 0 

fey. % 

^ 0 

0# 

o 

* 

f 

fp 

^ S' 0 

& o 

M' 

S.Hours 

0 * 
o | 

£p 0 

0 

r : 

S. 

O'- 0 - 

% 

iffc 0 

% 

% 

M 


o c- 

* 

CustSat 

o * 

o c- 

0 0 

*► 

c- c 

ec 

jik , 

O 

•0*0 So 

* * 

* .<3= 3 ^ 

0*S V>“ 0 

w 0 <? 

e 

B 

OrtTime 

€ l % 

SEJp . v 

c 

Iw v£s:= ’ o 
* 0 3> 

I 1 

■ 


0 * 
r> 

* - 0 

Aik- 

0 e 

0 § C ' 

BE 

' :< D 

c u 

o jfkfc® 

MU MM 

A * 

° 0 

i° 15 

s>. 

ft 0 

% 

cw^°' * 

0 

0 r .d 

% 

0 * 

Vl° 

C 2> :* 

® 0 G 

6* > * , 
m # o 
* 

SupCMM 

0 

& ° 
, 

¥ * 

■ 

■ 





OthCMM 



0 0 

8 I 


0.. 

&S * % 


& 

I. 

ROI 


Figure 3. Area Model Scatter Plots, Pairs of Candidate Measures 


23 


































A significant conclusion is therefore drawn that each output can be modeled as 
linear with respect to input and no transformations of data are necessary. Neither do 
decreasing returns to scale need to be accounted for in the DEA model. 



Figure 4. Plant Model Scatter Plots, Pairs of Candidate Measures 


2. Correlation Analysis 

Because of the linear nature of the data, analysis of the pairwise correlation 

coefficients between inputs and outputs gives a relative strength of this linear 

relationship. Tables 2 and 3 show the correlation between each measure for both model 

sets. Positive correlation between inputs and outputs indicates a strong candidate for 
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inclusion in a DEA model. However, strong correlation between inputs with other inputs 
and outputs with other outputs may indicate a redundancy in the measurement of one or 
more of these factors. Each factor, among its peers of inputs or outputs should measure a 
unique factor to the greatest extent possible. 

As suspected, the two input measures are highly correlated in both models, 
indicating that only one, not both, input measures should be chosen. Further, an arbitrary 
selection of one of these two inputs for inclusion in a model would not have a great 
impact on the overall result of the model. The real goal of scaling the input hours is not 
to affect the average outcome but to impact efficiency scores of offices individually. 



Hours 

SJiours 

CustSat 

Ontime 

MUMM 


OthCMM 

ROI 

Hours 

i 

.95 

-.13 

-.30 

.37 

.87 

.54 

.34 

S.hours 

.95 

1 

-.16 

-.30 

.48 

.81 

.50 

.26 

CustSat 

-.13 

-.16 

1 

.50 

.17 

-.06 

-.62 

.01 

Ontime 

-.30 

-.30 

.5 

1 

.17 

-.17 

-.30 

-.15 

MUMM 

.37 

.48 

.17 

.17 

1 

.29 

.03 

.29 

SupCMM 

.87 

.81 

-.06 

-.17 

.29 

1 

.43 

.21 

OthCMM 

.54 

.50 

-.62 

-.30 

.03 

.43 

1 

.21 

ROI 

.34 

.26 

.01 

-.15 

.29 

.21 

.21 

1 


Table 2. Correlation Coefficients for Measures, Area Model 



Hours 

S.hours 

CustSat 

Ontime 

MUMM 


OthCMM 

ROI 

Hours 

i 

.97 

.19 

,ii 

.36 

.49 

.64 

.40 

S.hours 

.97 

1 

.22 

.12 

.40 

.52 

.65 

.36 

CustSat 

.19 

.22 

1 

.20 

.11 

.16 

.28 

.05 

Ontime 

K91 

.12 

.20 

1 

.16 

.35 

.22 

.01 

MUMM 


.40 

.11 

.16 

1 

.13 

.01 

-.07 

SupCMM 

.49 

.52 

.16 

.35 

.13 

1 

.66 

.10 

OthCMM 

.64 

.65 

.28 

.22 

.01 

.66 

1 

.31 

ROI 

.40 

.36 

.05 

.01 

-.07 

.10 

.31 

1 

1 

fable 3. Correlation ( 

Coefficients for Measures, Plant Model 


The contract count measures (SupCMM and OthCMM) are also relatively highly 
positively correlated with each other. This indicates that offices that manage a large 
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number of supply contracts also manage a large number of other contracts. Although it is 
possible that an aggregate count of total contracts managed may provide a more 
meaningful result in the final efficiency analysis, aggregation also serves to dilute the 
effect of the major component of the CMM factors (supply contracts count) as a separate 
measure of output. For this reason, two CMM factors will remain. The similarities in the 
two data sets are substantial, however, key differences can be highlighted. 

a. Defense Contract Management Area Offices 

In the Area model, a high positive correlation exists between each input 
candidate and the two contract count measures. A somewhat lower correlation with 
MUMM as well as the ROI factor also exist. These four measures seem to be the core 
output measures in this model with respect to Hours and S.hours. A negative relationship 
exists for the Customer satisfaction and On-time rate. The Ontime correlation of -.3 
admits that most likely no positive relation does exist with the input. It is probable that 
an increase in hours has no effect on any contractor’s ability to meet his contracted 
schedule. 

In fact, the results of a two tailed test of the null hypothesis of zero 
correlation on each of the correlation coefficients, (at a .05 level of significance), show 
that most likely no correlation exists between ROI, CustSat, or OnTime with either 
Hours or S.Hours. (The other measures show strong evidence of correlation.) These 
three uncorrelated measures constitute half of the potential outputs in the model. 
However, this result is not completely unexpected as the current staffing levels have been 
assigned with the regression model (discussed in Chapter II) utilizing only workload as 
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the modeled variable. The three uncorrelated measures have never been "rewarded" with 


additional staff resources in the past. It is possible that after implementation of a model 
for staff resourcing that does reward these measures, correlation will increase. In any 
case, the inclusion of customer satisfaction rating and on-time factor will simply serve as 
another factor for the offices with a high scores in these metrics to take advantage of in 
the DEA methodology. Additionally, they are the only qualitative measures available, 
and as such, must be included in the model formulation. Chapter IV will consider some 
measures to limit their overall effect on the efficiency scores calculated. 

b. Plant Representative Offices 

Relationships in the Plant Representative model are much more clear. 
Every input and output relationship is indeed positive although those relationships vary 
greatly in their magnitude. The greatest correlation exists in what appear to be the core 
measures with respect to hours and scaled hours: MUMM, SupCMM, OthCMM, and 
ROI. But a different correlation than in the area model exists with customer satisfaction 
rate. This may be in part due to the lower reporting rate among plant offices for this 
factor (82% in plants versus 94% for area offices) and higher average score. 

The on-time rate factor, while positively correlated with scaled hours at 
.12 or with hours at .11 is still the lowest, by far, of all other measures. The results of a 
two tailed test of zero correlation on each of the correlation coefficients, (at a .05 level of 
significance), show that likely no correlation exists between the CustSat or OnTime 
measures with either Hours or S.Hours. Unlike the Area model, ROI does show evidence 
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of correlation. As in the Area Model, these factors may be included in the model, with 
restrictions on its overall effect. 

c. The Effect Of Management 

As noted earlier, CAOs have been resourced according to a regression 
model utilizing workload against staffing. It is likely that this method of resourcing has 
served to undermine the correlation of output measures not related to that regression 
model (on-time rate and customer satisfaction). Further, even though measures of 
quality were not required in the regression model, they are absolutely necessary in an 
efficiency model where outputs are assigned rather than under the control of office 
management. For these reasons, quality measures, regardless of their negligible 
statistical correlation with the input must be included in both models. 


C. DEA MODEL FORMULATION 

The linearity of every output / input plot implies that a constant return to scale 
relationship exists among them. Evidence suggests that a CCR DEA model formulation 
best fits the DCMC data for efficiency evaluation. The general CCR model general 
formulation shown in Chapter II can be reformulated as a linear program with the specific 
parameters and indices from the DCMC data as follows: 
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Indices: 


i Output Types (SupCMM, OthCMM, MUMM, ROI, CustSat, OnTime) 

j Input Types (Hours or S.Hours) 

k DMUs (Contract Administration Offices (1 to 34 for Plant Model and 1 to 31 for 

Area Model)) 

Data: 


Xkj 

Value of input j for DMU k 

ya 

Value of output i for DMU k 

8 

A small constant. 

Variables: 

v j 

Weight on input j (unitless) 

w, 

Weight on output i (unitless) 

Formulation: 

For each DMU o, o = 1,2,...K 


Maximize z c = X y 0 i w* 

i 

Subject to: 


£ (Xkj Vj ) - £ (yid Wi) <0, k= 1,2,...K 

j i 


Z Xoj Vj = 1 

j 


Vj > e, j= 1,2,...J 

Wj>8, i= 1,2,...I 


Figure 5. Linearized CCR Formulation (Norman, 1991, p. 236) 


A total of 65 linear program solutions are required for the two models for DCMC, 
one for each DMU. 


D. CONCLUSIONS 


The DEA models for both Plant and Area offices will be CCR DEA models that 

contain either total hours or scaled hours as the input. Comparisons between the two 

■ 

■ 

input factors will be considered after model formulation. Outputs in each model will be 
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CustSat, SupCMM, OthCMM, MUMM, ROI and OnTime. Even though correlation 
analysis of the “quality” type measures do not suggest inclusion in the model, they are 
absolutely necessary for implementation in this model because all other outputs are 
assigned rather than under the control of the individual office management team. The 
qualitative measures are required to ensure that the assigned outputs are being performed 
well (Ganley, 1992, p.36). 

The following chapter will outline the final model formulation for each office 
type and discuss the results of the CCR DEA models for both plant and area models. 

Also suggested improvement strategies for individual inefficient offices will be analyzed. 
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IV. MODEL RESULTS AND ANALYSIS 


The two DEA efficiency models (Plant Office Model and Area Office Model) are 
created with one input parameter and six output parameters. Because of the general 
linear relationship between the input and output values, the DEA formulation that 
assumes constant returns to scale was used to express an optimally weighted ratio of 
outputs to inputs. The CCR formulation, outlined in Chapters II and El produced optimal 
efficiency scores for each evaluated Contract Administration Office. 

Here, Decision Making Units (DMU) are Contract Administration Offices, the set 
of inputs is either Hours or S.Hours, and the set of outputs are SupCMM, OthCMM, 
MUMM, ROI, CustSat, and OnTime. The 65 fractional programs, (34 for Plant Model 
and 31 for Area Model), which can be equivalently expressed as linear programs (LP), 
were solved with the General Algebraic Modeling System (GAMS), utilizing the OSL 
solver. 

In this chapter, a final formulation of the two models is presented after analyzing 
two unanswered questions from the previous chapter: What is the proper input measure? 
How can limits on the un-correlated metrics be imposed? Once the models are finally 
formulated, analysis of efficiency scores and insights into improvements for inefficient 
offices can be made. 

A. FINAL MODEL FORMULATION 

1. Effect Of Risk Factor Scaling 

The effect of scaling hours by risk factor was unknown prior to modeling but 
expected to slightly lower efficiency in low risk offices and slightly raise efficiency in 
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those with high risk while not appreciably changing the average efficiency score over all 
offices. This should assure that offices with high risk are not penalized because their 
workload requires a greater input expenditure by its very nature and that low risk offices 
are not given an unfair advantage. By scaling the input hours, the intended result is that 
those offices with low risk will have to "work harder" to achieve efficiency than those 
with high risk. 

The resulting efficiency scores show decisively that this is the case. A 
comparison of the aggregated mean efficiency score for each model for each different 
input, shows that a very small change took place (see Table 4). The change from un¬ 
sealed hours to scaled hours in the Area Model resulted in a slight decrease in average 
score from 76.36% to 75.47% while in the Plant Model a slight increase from 72.88% to 
74.78% was shown. The conclusion is that the scaling factor has little impact on the 
average efficiency score. 

In examining the score changes for each individual office between the two 
models, the greatest impact of scaling can be seen in the extreme cases. The five lowest 
risk and five highest risk offices show the most dramatic score change. 
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Five Offices with 
Lowest Risk 

Score 

no 

Scaling 

Score 

with 

Scaling 

change 

% change 

Area Office 31 

100.00 

100.00 

0.00 

0.00% 

Area Office 23 

45.88 

34.11 

-11.77 

-25.66% 

Area Office 19 

100.00 

100.00 

0.00 

0.00% 

Area Office 25 

78.39 

55.14 

-23.25 

-29.66% 

Area Office 20 

98.37 

74.47 

-23.90 

-24.29% 

Five Offices with 
Highest Risk 

Score 

no 

Scaling 

Score 

with 

Scaling 

change 

% change 

Plant Office 17 

49.48 

67.20 

17.72 

35.81% 

Plant Office 18 

56.73 

73.21 

16.48 

29.06% 

Plant Office 23 

55.06 

61.23 

6.17 

11.20% 

Plant Office 9 

79.45 

90.41 

10.96 

13.80% 

Plant Office 21 

100.00 

100.00 

0.00 

6.00% 


Table 4. Risk Factor Scaling 


Note that offices in Table 4 with low risk that were rated efficient in the un-scaled 
model remained so in the scaled model. The strength of these offices in outputs is 
evident in their ability to overcome the scaling up of its input hours with risk factor and 
remain efficient in comparison to all other offices. In the case of the high risk office that 
was efficient in the un-scaled model, a further decrease (or scale down) of its input hours, 
has no further effect as it is already able to maximize its efficiency score. Also of interest 
is the fact that the 5 lowest risk offices are all area offices and the 5 highest risk offices 
are all plant offices. The graphs below (Figures 6 and 7) show in detail the individual 
score changes as impacted by risk factor scaling of the model. 
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igure 6. Area Model Efficiency Scores, Scaled Hours vs. Un-scaled hours 


Plant Model 



Efficiency With Unsealed Hours Input 


Figure 7. Plant Model Efficiency Scores, Scaled Hours vs. Un-Scaled Hours 







The evidence supports the hypothesis that scaling by risk factor has the intended 
effect of equalizing the inputs for all offices based on inherent risk. For this reason, only 
models with the scaled input hours will gain further consideration for employment and 
refinement. 

2. Effect Of Variable Constraints On Efficiency Scores 

A great advantage of DEA methodology is the ability for each unit to be evaluated 
on its strengths. If the unit has only one strong parameter in a field of six, it is still 
possible to receive an efficiency score of one (100%). In this case, however, a few 
parameters did not display a positive relationship with the input parameter. An efficient 
rating for an office utilizing only one of those less strongly correlated factors should be 
limited in order to gain the greatest insight from the eventual scores. The problem is 
easily visualized when one assumes the worst case: An office with very few contracts to 
manage, with small dollar value, but with a very good OnTime and CustSat rating can 
certainly be evaluated as "efficient". In fact, the results of the models bear this out. 

In the Area Model, without constraints on the maximum values of the weights, 
five of 31 offices rely on one of the weakly correlated parameters for more than 50% of 
their efficiency rating. In one case, fully 94% of the output weight (variable value of .94) 
is on the customer satisfaction rating. The effect on all offices are shown at Figure 8 with 
the most constraint sensitive offices labeled. 

The Plant Model felt a significantly greater impact on efficiency scores with the 
imposition of variable constraints. Plant Model offices tend to rely on fewer parameters 
than do the Area Model Offices. This was expected because area offices manage 
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contracts from a wide variety of contractors and even wider range of contract kinds and 
types resulting in an efficiency score that combines many factors. In comparison, the 
plant offices, often with only one contractor to service, tend to be strong in only one or 
two performance measures. On average. Area Model offices used 3.5 output parameters 
in their efficiency score while Plant Model offices used only 2.6. As a result, the impact 
of constraints on two parameters is magnified in the Plant Model. Of concern, however, 
is the fact that three offices in the Plant Model relied completely on the customer 
satisfaction rating and fully 12 offices (35% of all plant offices) used the customer 
satisfaction rate for more than 75% of their overall efficiency score. 

Area Model 

Unconstrained vs. Constrained 



Efficiency With Constraints on 3 Parameters 


Figure 8. Area Model Constraint Impact on Efficiency Scores 
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Clearly, a limit on these factors is in order. An upper limit of 50%, 33%, or 25% 
on the statistically un-correlated parameters is imposed and has the effect of drawing 
down the score of the offices heavily dependant on them and forcing more reliance on the 
more highly correlated metrics. 


Plant Model 

Unconstrained vs. Constrained 



0 10 20 30 40 50 60 70 , 80 90 100 

• 50% Limit D33% Limit +25% Limit 
Efficiency With Constraints on 2 Parameters 

Figure 9. Plant Model Constraint Impact On Efficiency Scores 


With the imposition of constraints, it is possible for offices to either receive the 
same score as in the unconstrained model or a lower score. In some cases this lower 
score is dramatic; the result of an over reliance on the un-correlated parameters for 
efficiency rating. The effect of the 25% and 33% constraint appear to be overly 
restrictive. The 50% constraint has the intended effect of significantly reducing the 


37 




efficiency score for offices that rely too heavily on those metrics. The more restrictive 
constraints begin to adversely impact offices that do not seem overly dependant on those 
parameters for their efficiency score. The 50% upper limit constraint on CustSat, 
OnTime, and ROI will be imposed in the Area Model, the same constraint will placed 
only on CustSat and OnTime in the Plant Model. 

3. The Final Model 

The two final pieces of the model formulation are now in place. The models are 
both constant returns to scale models that utilize total hours scaled by risk factor as an 
input, all six outputs measures, with constraints on the un - correlated parameters 
restricting their weight to less than 50% of each overall efficiency score. Analysis of the 
efficiency scores and an examination of improvement methods for inefficient offices can 
now be examined. 

B. EFFICIENCY SCORE SUMMARY 

1. Overview Of Results 

Efficiency scores for all offices are at Appendix A. The results of each model are 
summarized in the following tables and figures. 



Area 

Plant 

Number Efficient 

7 

8 

Mean 

74.71% 

66.58% 

Median 

72.07% 

69.99% 

Standard Deviation 

21.46% 

27.30% 

Range 

80.67% 

99.38% 

Minimum 

19.32% 

0.61% 

Maximum 

100% 

100% 


Table 5. Summary Statistics 
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Examination of the summary scores leads one to assume that the model results 
were quite similar. However, closer inspection of score frequency diagrams tells a 
different story. While most of the Area Model's offices have scores above 50%, the Plant 
Model has a significant amount of low scoring offices. 
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Figure 10. Area Model Score Frequency Stem and Leaf Display 
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Figure 11. Plant Model Score Frequency Stem and Leaf Display 


Inspection of the data for these four poor performers reveals insight into their low 

efficiency score. While in all cases, their input hours value was near the average of all 

other plant offices, all four reported zero for their MUMM statistic. That is: the value of 

their contracts managed was zero. Because MUMM data is only collected for one kind of 
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contract (Systems Acquisition), this fact alone is not terribly unusual. However, closer 
inspection of the input data shows hours logged specifically for Systems Acquisition 
contracts. This leads to the assumption that a Systems Acquisition contract value greater 
than zero, no matter how small, should be in the MOCAS database for these offices. 
Additionally, their contract count statistics, while not zero, were very low compared to 
other offices. The lower value of the quantitative output measures for these four offices 
(MUMM, SupCMM, and OthCMM) coupled with the imposed constraint on the 
qualitative measures ensured their low score. 

It seems clear that some type of data collection error is in evidence in either the 
PLAS or MOCAS database for these few offices. Fortunately, this does not undermine 
the methodology. While an artificially high measure in the data may perturb the results 
of the model, incorrect low scores will not. In any case, inferences about the efficiency 
of these four offices are difficult to ascertain. 

2. Quantitative vs. Qualitative Measures 

Plant and Area offices generally rely on different subsets of outputs in optimizing 
their efficiency scores. The Area Model consistently relied on quantitative measures to 
produce efficiency scores for its office. The early data analysis in Chapter II certainly 
showed that a significant relationship exists with these measures while the qualitative 
measures failed to show any correlation was present in hypothesis testing. In Figure 10, 
The portion of quantitative measures making up the efficiency score of each offices is 
plotted against the total efficiency score. (For example, the portion of an efficiency score 
of 15% may be made of 50% quantitative measures and 25% qualitative measures.) 
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Those offices lying along the "100% Dependence" line base their entire efficiency score 
on the quantitative measures. The farther offices lie from that line, the more they are 
dependant on the qualitative measures. Figure 13 shows the Area Model qualitative 
measures vs. their overall efficiency score. This graph is the complement or "mirror" of 
Figure 12. Those offices with a high dependence on qualitative measures are now nearer 
the 100% dependence line. 


Area Model 

Efficiency vs. Quantitative Measures 



Figure 12. Area Model Efficiency Scores, Quantitative Contribution 
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Area Model 



Weighted Sum of CustSat, ROI, OnTime 
Figure 13. Area Model Efficiency Scores, Qualitative Contribution 

Inspection of the same plots with the Plant Model shows that quite a different 
parameter combination is at work. In those offices that rely heavily on either quantitative 
or qualitative measures, more favor the quality type measures. However, no clear trend is 
admitted in the Plant Model. It seems that the Plant offices use a variety of parameter 
combinations to calculate efficiency scores. 
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Figure 14. Plant Model Efficiency Scores, Quantitative Contribution 



Figure 15. Plant Model Efficiency Scores, Qualitative Contribution 
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C. IMPROVEMENTS FOR THE INEFFICIENT 


For an office to improve its efficiency score it must either reduce inputs, increase 
outputs, or both. By examining an inefficient office's reference set, improvement options 
can be identified. 


1. Reference Sets 

The reference set is the set of efficient units (offices) that an inefficient unit has 
been most directly compared with when calculating its efficiency rating. It contains the 
efficient units which have the most similar input and output orientation as the inefficient 
unit does. The reference set offices should therefore provide examples of good operating 
practice for it to emulate. 

Area Office 29, an inefficient office, has four efficient offices in its reference set: 
Offices 5,7,9, and 31. As detailed in Table 6, each reference set office, using Office 29's 
optimal weights and not their own, can still achieve an efficiency score of 100%. 


Office 


S.Hours 

CustSat 

OnTime MUMM SupCMM OthCMM 

ROI 

Score 

5 

Value 

148225 

5.74 

52.35 

3921 

46289 

24084 

13135 



Weight*Value 

1.19 

0 

21.25 

60.88 

0 

11.90 

26.40 

100 

7 

Value 

41672 

6.00 

70.45 

256 

18819 

1334 

2 



Weight* Value 

0.33 

0 

28.59 

3.98 

0 

0.66 

0.01 

100 

19 

Value 

143907 

5.85 

61.90 

5330 

52008 

7973 

1640 



Weight*Value 

1.15 

0 

25.12 

82.75 

0 

3.94 

3.30 

100 

31 

Value 

94051 

5.65 

64.17 

3102 

15385 

904 

226 



Weight*Value 

0.75 

0 

26.04 

48.17 

0 

0.45 

0.46 

100 

29 

Value 

124816 

5.71 

74.61 

3428 

39586 

7428 

2283 



Weight*Value 

1.00 

0 

30.28 

53.22 

0 

3.67 

4.59 

91.76 












Table 6. Reference Set Comparison 


Inspection of the values in the reference set allows Office 29 to compare itself 

with offices that operate under similar conditions. Emulating one these offices guides 
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Office 29 in the direction of efficiency. From the table. Office 29 is fairly consistent with 
offices in its reference set. Because of it already relatively high score (91.76%), no 
obvious poor performance in this office is readily seen. However, it is possible that a 
modest improvement could be made by increasing ROI to align more closely with 
Office 5. 

An interesting result of having only one input measure for each model is the fact 
that by holding the current output factors constant, the efficiency score can be increased 
by a corresponding decrease in S.Hours. Simply put, an input level that is 91.76% of the 
current level would increase the efficiency score to 100%. Put another way. Area office 
29 can become efficient by reducing its input hours by 8.24% or about 54 thousand 
hours. 


2. Efficient Frontiers 

By collapsing the outputs into two groups, the quantitative and qualitative, the 
efficient frontier can be seen in two dimensions. In Figure 16, Office 29 appears close to 
the frontier. Area Office 29 can increase efficiency by moving right, increasing its 
qualitative factors, or by moving up, increasing its quantitative factors. The most direct 
route to the efficient frontier is the path from the origin through point 29 onto the frontier. 
This direction of improvement shows the most rapid route to efficiency. However, in 
DCMC's case, quantitative measures are not under the control of local management. 

Most likely, this office can only move right to the frontier to achieve efficiency. As a 
comparison, the efficient frontier for the Plant model is at Figure 17. 
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OthCMM/Weighted Input Weighted Sum of MUMM, SupCMM, OthCMM/ 


Area Model 
Hiflcient Frontier 

Qualitative vs. Quantitative Measures 



Figure 16. Area Model Efficient Frontier 


Plant Model 
Efficient Frontier 

Qualitative vs. Quantitative Measures 



Figure 17. Plant Model Efficient Frontier 
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D. EFFICIENT UNITS AND OVERHEAD COST 

In Chapter H, an environmental factor, overhead cost, was evaluated as a potential 
input or output. This measure showed the dollar value cost per hour for each office, that 
was not strictly related to the core tasks. These types of costs included training costs and 
travel time costs, among others. Unfortunately, this measure did not have the attributes 
to contribute to the model as either an input or output. However, overhead cost may 
contribute to this study by showing an overhead cost trend in efficient offices. If such a 
trend exists, inefficient offices, by means of comparison, can determine if their overhead 
costs are too small or too large. Too little overhead cost may indicate that not enough 
time is spent in organization-sustaining work such as training. Too much may show that 
staff is distracted from its core tasks of administering contracts. 

Figures 18 and 19 show efficiency scores plotted with overhead costs. The 
plotted means (dashed lines) for both efficient offices and for all offices seem to show 
that efficient offices have a somewhat smaller average overhead cost associated with 
them. However, in the Plant Model, a level .05 Wilcoxon rank-sum test strongly rejects 
the null hypothesis that the efficient office mean overhead cost is the same as the mean 
for all offices. This suggests that in the Plant Model, efficient offices have a significantly 
lower than average overhead cost. Such strong evidence does not exist in the Area Model 
even though the average overhead cost is higher for inefficient offices. In general, an 
overhead cost significantly higher than the average efficient office's may be an indicator 
of inefficiency, although some exceptions are present. 
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Figure 18. Plant Model Overhead Costs 
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Figure 19. Area Model Overhead Costs 
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E. CONCLUSION 


This chapter commenced with the final formulation of DEA models for DCMC. 
All the previous data analysis and model investigation from Chapters I through HI led to 
a DEA model that assumes constant returns to scale but with added constraints on some 
variables. Constraints were imposed to limit the effect of those output parameters that 
displayed little or no correlation with the input parameter. 

Scores were then analyzed to determine by what means inefficient offices could 
become efficient. Finally, by observing the overhead costs associated with each office, a 
trend was found to show inefficient offices yet another possible avenue to improve 
efficiency score. Chapter V will discuss specific recommendations for further study, as 
well as recommendations for DCMC should these models be implemented to assist in 
policy decisions. 
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V. RECOMENDATIONS 


One goal of this thesis is to determine the efficiency of all 65 CAOs in DCMC. 
Only 15 of these offices are shown to be efficient. It is important to remember, however, 
that each office is evaluated in the very best possible light with respect to the given model 
parameters. If the assumption that the six modeled outputs and one modeled input make 
up the vast majority of resources and products within DCMC, then the resultant DEA 
efficiency score is the best possible score for each office. 

A. MODEL VULNERABILITY 

The model in its current form is vulnerable to data manipulation by individual 
offices. Because some of the data elements originate within each CAO, those elements 
are not subject to auditing by higher level management. Especially vulnerable to internal 
manipulation is the ROI factor and the total Hours figure. Each are reported directly by 
the CAOs. While not to imply that wholesale manipulation of the data by the individual 
offices would arise, certainly some system of auditing those reports would be prudent. In 
fact, the problem with the Hours figure is easily solved. A comparison of the total Hours 
figure with the number of employees in each office shows these two measures to be 
nearly identical. Modeling with the number of employees provides nearly identical DEA 
efficiency scores. Hours were used in these models because DCMC believes that the 
PLAS database gives a more accurate view of total input. 

The other measures used in the study are not generated at the CAO level and are 
sufficiently robust as to need no further manipulation. 
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B. DATA COLLECTION CONCERNS 

The quality of the customer satisfaction rating is questionable due to the low 
report rate by serviced contractors. In order to make CustSat a higher quality measure, a 
concerted effort by management, which may ultimately require a new approach to 
customer satisfaction data collection, is certainly in order. Surely a bottom line quality 
measure is absolutely essential in a model such as this and customer satisfaction has 
potential to be that measure. 

DEA literature emphasizes that a causal relationship between inputs and outputs 
must be present (Norman, 1991, p. 178). Because no relation between input hours and 
these qualitative measures is evident present, management should implement systems to 
reward, or at least emphasize these measures as the key quality metrics for the entire 
organization. Once these new policies have been established, increased correlation 
between the input and these measures are expected. 

C. INTERPRETING THE RESULTS 

The result that 50 offices of 65 in DCMC have an efficiency score less than 100% 
does not immediately imply that an automatic reduction in personnel strength is in order 
for all 50 offices. Certainly, a best approach is to examine an inefficient office’s 
reference set, determine obvious improvement opportunities, and work to improve 
efficiency. Ultimately, however, some offices may indeed require staffing reduction. 

Further, this study did not take into account any type of "baseline" personnel 

resource requirement. If we assume that every office needs a minimum number of 

employees to accomplish daily routine missions, regardless of the workload assigned, it is 

a fair argument that the DEA model should only model those hours above the baseline 
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figure. This baseline figure may account for the poor showing among a few small Plant 
Offices. 

D. CONCLUSION 

The DEA models presented in this study provide a valuable management tool to 
assist DCMC in determining personnel resource requirements. However, these models 
have limitations. Used in conjunction with the current system of resource forecasting (a 
regression model), offices with significant resource issues and problems can be quickly 
identified. The great added value with the DEA models is the new visibility over each 
office's strengths and weaknesses. The rapid identification of these weaknesses allows 
management to focus on problems and solve them quickly without a great time 
investment searching for possible causal factors. 
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APPENDIX. EFFICIENCY SCORES 


Each Model (Plant Office and Area Office) was solved with various formulations. 
The resulting scores below detail the three final formulations discussed in the text. 

Model 1 uses the un-scaled input of Hours with all six outputs. No constants are 
placed on any output parameters. 

Model 2 uses the scaled hours input (S.Hours). The outputs are identical to 
Model 1. 

Model 3 uses the S.Hours as input and constrains the qualitative output measures 
(CustSat, OnTime, and ROI) to less than 50% of each efficiency score. This model was 
selected for further analysis. 
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CAO 

MODEL 1 

MODEL 2 

MODEL 3 

1 

45.51 

30.64 

19.33 

2 

71.87 

74.12 

74.12 

3 

100.00 

100.00 

100.00 

4 

60.40 

59.81 

59.81 

5 

100.00 

100.00 

100.00 

6 

62.42 

64.68 

64.68 

7 

100.00 

100.00 

100.00 

8 

100.00 

100.00 

100.00 

9 

75.39 

99.84 

99.84 

10 

87.29 

89.31 

89.31 

11 

57.59 

58.87 

57.08 

12 

69.10 

75.16 

75.16 

13 

73.67 

74.10 

74.10 

14 

70.12 

69.07 

67.38 

15 

62.74 

65.83 

65.83 

16 

53.71 

60.29 

60.29 

17 

79.18 

71.47 

71.47 

18 

92.44 

97.60 

97.60 

I 19 

100.00 

100.00 

100.00 

20 

98.37 

74.47 

68.88 

21 

62.50 

50.00 

50.00 

22 

100.00 

100.00 

100.00 

23 

45.88 

34.11 

34.11 

24 

78.57 

84.80 

84.80 

25 

78.39 

55.14 

55.14 

26 

47.44 

47.77 

47.77 

27 

70.31 

67.03 

63.97 

28 

72.33 

72.07 

72.07 

29 

80.14 

91.76 

91.76 

30 

71.68 

71.54 

71.54 

31 

100.00 

100.00 

100.00 


Table A-l. Area Model Efficiency Scores 
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CAO 

Model 1 

Model 2 

Model 3 

1 

100.00 

100.00 

100.00 

2 

51.72 

57.49 

13.94 

3 

62.11 

61.39 

59.84 

4 

66.42 

64.55 

52.68 

5 

100.00 

100.00 

100.00 

6 

45.29 

45.15 

44.11 

7 

66.78 

70.34 

70.34 

8 

74.41 

73.49 

69.66 

9 

79.45 

90.41 

86.60 

10 

56.33 

56.32 

56.32 

11 

45.85 

52.46 

52.07 

12 

39.82 

45.11 

45.11 

13 

86.96 

96.25 

88.63 

14 

92.36 

82.70 

75.64 

15 

82.63 

71.15 

71.15 

16 

76.10 

77.24 

77.24 

17 

49.48 

67.20 

50.39 

18 

56.73 

73.21 

73.21 

19 

100.00 

100.00 

100.00 

20 

100.00 

100.00 

100.00 

21 

100.00 

100.00 

100.00 

22 

56.07 

58.85 

58.85 

23 

55.06 

61.23 

60.60 

24 

68.93 

68.35 

45.25 

25 

95.25 

91.88 

61.39 

26 

0.53 

0.62 

0.62 

27 

81.53 

96.13 

51.24 

28 

100.00 

100.00 

100.00 

29 

100.00 

100.00 

100.00 

30 

81.84 

92.40 

75.54 

31 

100.00 

100.00 

100.00 

32 

65.78 

57.98 : 

27.61 

33 

85.88 

84.02 

83.32 

34 

54.53 

46.53 

12.72 


Table A-2. Plant Model Efficiency Scores 
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